Highly scalable distributed connection interface for data capture from multiple network service sources

ABSTRACT

A highly scalable distributed connection interface for data capture from multiple network service sources, comprising a connector module wherein, the connector module retrieves a plurality of data from a plurality of network data sources; employs a plurality of application programming interface routines to communicate with the plurality of data sources; accepts a plurality of analysis parameters and control commands directly from human interface devices or from one or more command and control storage devices; and specifies the action or actions to be taken on the retrieved data.

CROSS-REFERENCE TO RELATED APPLICATIONS

Application No. Date Filed Title Current Herewith HIGHLY SCALABLEDISTRIBUTED application CONNECTION INTERFACE FOR DATA CAPTURE FROMMULTIPLE NETWORK SERVICE SOURCES Is a continuation of: 16/660,727 Oct.22, 2019 HIGHLY SCALABLE DISTRIBUTED CONNECTION INTERFACE FOR DATACAPTURE FROM MULTIPLE NETWORK SERVICE SOURCES which is a continuationof: 15/229,476 Aug. 5, 2016 HIGHLY SCALABLE DISTRIBUTED Patent IssueDate CONNECTION INTERFACE FOR DATA 10,454,791 Oct. 22, 2019 CAPTURE FROMMULTIPLE NETWORK SERVICE SOURCES which is a continuation in-part-of:15/206,195 Jul. 8, 2016 ACCURATE AND DETAILED MODELING OF SYSTEMS WITHLARGE COMPLEX DATASETS USING A DISTRIBUTED SIMULATION ENGINE which is acontinuation in-part-of: 15/186,453 Jun. 18, 2016 SYSTEM FOR AUTOMATEDCAPTURE AND ANALYSIS OF BUSINESS INFORMATION FOR RELIABLE BUSINESSVENTURE OUTCOME PREDICTION which is a continuation in-part-of:15/166,158 May 26, 2016 SYSTEM FOR AUTOMATED CAPTURE AND ANALYSIS OFBUSINESS INFORMATION FOR SECURITY AND CLIENT-FACING INFRASTRUCTURERELIABILITY which is a continuation in-part-of: 15/141,752 Apr. 28, 2016SYSTEM FOR FULLY INTEGRATED Patent Issue Date CAPTURE, AND ANALYSIS OFBUSINESS 10,860,962 Dec. 8, 2020 INFORMATION RESULTING IN PREDICTIVEDECISION MAKING AND SIMULATION which is a continuation in-part-of:15/091,563 Apr. 5, 2016 SYSTEM FOR CAPTURE, ANALYSIS AND Patent IssueDate STORAGE OF TIME SERIES DATA FROM 10,204,147 Feb. 12, 2019 SENSORSWITH HETEROGENEOUS REPORT INTERVAL PROFILES and is also a continuationin-part-of: 14/986,536 Dec. 31, 2015 DISTRIBUTED SYSTEM FOR LARGE PatentIssue Date VOLUME DEEP WEB DATA 10,210,255 Feb. 19, 2019 EXTRACTION andis also a continuation in-part-of: 14/925,974 Oct. 28. 2015 RAPIDPREDICTIVE ANALYSIS OF VERY LARGE DATA SETS USING THE DISTRIBUTEDCOMPUTATIONAL GRAPH the entire specification of each of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is in the field of use of computer systems inbusiness information management, operations and predictive planning.Specifically, the use of a highly scalable, distributed, and self-loadbalancing connection interface programmed to capture information from awide range of network service sources and then format that informationfor tightly specified downstream business information system uses.

Discussion of the State of the Art

Over the past decade, the amount of financial, operational,infrastructure, risk management and philosophical information availableto decision makers of a business from such sources as ubiquitous sensorsfound on a business's equipment or available from third party sources,detailed cause and effect data, and business process monitoring softwarehas expanded to the point where the data has overwhelmed corporateexecutives' abilities to follow all of it and certainly to interpret andmake meaningful use of that available data in a given businessenvironment. In other words, the torrent of information now available toa corporate decision maker or group of decision makers has far outgrownthe ability of those in most need of its use to either fully follow itor reliably use it. Failure to recognize important trends or becomeaware of information in a timely fashion has led to highly visible,customer facing, outages at NETFLIX™, FACEBOOK™, and UPS™ over the pastfew years, just to list a few.

There have been several developments in business software that havearisen with the purpose of streamlining or automating either businessdata analysis or business decision process. PLANATIR™ offers software toisolate patterns in large volumes of data, DATABRICKS™ offers customanalytics services, ANAPLAN™ offers financial impact calculationservices and there are other software sources that mitigate some aspectof business data relevancy identification, analysis of that data andbusiness decision automation, but none of these solutions handle morethan a single aspect of the whole task. This insinuates the technologybeing used in the decision process as one of the variables as data fromone software package often must be significantly and manuallytransformed to be introduced into the software for the next analysis, ifappropriate software exists. This step is both inefficient use of humanresources and has potential to introduce error at a critical processpoint.

There has also been a great proliferation in the use of network basedservice companies offering solutions for such business functions ascustomer relationship management, world event news sourcing, market newssourcing and analysis, infrastructure monitoring, human resourcemanagement, business real estate conditions, and government activityinformation sourcing from the world. This only serves to add to theoverload of information described above, and, to be of use, must becarefully analyzed by any business information management systempurporting to provide reliable predictions. A robust connectioninterface to all network service sources of business interest must beprovided.

Currently, there are a small number of scriptable data capture and sortinterfaces such as: Zapier and IFTTT, both able to connect to a numberof network data sources. However, these offerings possess only verylightweight logic options for moving the captured data into specificcategories or transformation pathways which greatly limit theirusefulness in complex business situations often encountered. Another,Open Source, capture engine, Sparkta is focused on streaming aggregationand fails to provide flexibility for routinely supporting event-drivenpolling, in addition to, passive stream monitoring of third-party APIsand similar operations needed by a business operating system.

What is needed is a fully integrated system that retrieves businessrelevant information from many disparate and heterogeneous sources usinga scalable, expressively scriptable, connection interface, identifiesand analyzes that high volume data, transforming it to a business usefulformat and then uses that data to drive an integrated highly scalablesimulation engine which may employ combinations of the system dynamics,discrete event and agent based paradigms within a simulation run suchthat the most useful and accurate data is obtained and stored for theneeds of the analyst. This multimethod information capture, analysis,transformation and outcome prediction system forming a “businessoperating system.”

SUMMARY OF THE INVENTION

Accordingly, the inventor has developed a highly scalable distributedconnection interface for data capture from multiple network servicesources. The connection interface is designed to enable simple toinitiate, performant and highly available input/output from a largeplurality of external networked service's and application's applicationprogramming interfaces (API) to the modules of an integrated predictivebusiness operating system. To handle the high volume of informationexchange, the connection interface is distributed and designed to bescalable and self-load-balancing. The connection interface possessesrobust expressive scripting capabilities that allow highly specifichandling rules to be generated for the routing, transformation, andoutput of data within the business operating system. Incoming data maybe received by passive stream monitoring, or by programmed, event ortime driven download of network service information to name just twopossibilities. Output may be direct tabular display of raw ortransformed data, graphical or derived graphical display, such assimulation presentation, either with or without persistence. Data may bepersistently stored in any of several data stores for which theconnection interface has internal API routines.

According to a preferred embodiment of the invention, a system forhighly scalable distributed connection interface for data capture frommultiple network service sources has been devised and reduced topractice. A connector module stored in a memory of and operating on aprocessor of a computing device wherein, the connector module: retrievesa plurality of data from a plurality of network data sources; employs aplurality of application programming interface routines to communicatewith the plurality of data sources; accepts a plurality of analysisparameters and control commands directly from human interface devices orfrom one or more command and control storage devices; and specifies theaction or actions to be taken on the retrieved business data;

According to another embodiment of the invention, a system for highlyscalable distributed connection interface for data capture from multiplenetwork service sources has been devised and reduced to practice. Aconnector module retrieves at least a portion of the data by continuousmonitoring of information streams released by the network data sources.At least a portion of the streaming data may be isolated based upon useof filters. At least a portion of the data is retrieved from networkdata sources based upon an event trigger. At least a portion of the datais retrieved from network data sources based upon a time dependenttrigger. At least a portion of the retrieved data is transformed by theconnector module into a format useful for a pre-determined purpose. Atleast a portion of the retrieved data is routed to other modules in abusiness operation system for transformation into a format useful for apre-determined purpose. At least a portion of the retrieved data isdisplayed and discarded. At least a portion of the retrieved data ispersistently stored.

According to a preferred embodiment of the invention, a method forhighly scalable distributed connection interface for data capture frommultiple network service sources comprising the steps of: a) retrievinga plurality of data from a plurality of network data sources using aplurality of network data source specific application programminginterface routines present in a connector module stored in a memory ofand operating on a processor of a computing device; and b) routing theplurality of data to a plurality of modules comprising a businessoperating system based upon data specific parameters present in aconnector module stored in a memory of and operating on a processor of acomputing device.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. One skilled inthe art will recognize that the particular embodiments illustrated inthe drawings are merely exemplary, and are not intended to limit thescope of the present invention.

FIG. 1 is a diagram of an exemplary architecture of a distributedoperating system according to an embodiment of the invention.

FIG. 2 is a diagram of an exemplary architecture of a connector moduleand related modules according to an embodiment of the invention.

FIG. 3 is a flow diagram of the operation of an exemplary connectormodule according to an embodiment of the invention.

FIG. 4 is a process flow diagram of a method for the receipt, processingand predictive analysis of streaming data using a system of theinvention.

FIG. 5 is a flow diagram for a linear transformation pipeline systemwhich introduces the concept of the transformation pipeline as adirected graph of transformation nodes and messages according to anembodiment of the invention.

FIG. 6 is a flow diagram for a transformation pipeline system where oneof the transformations receives input from more than one source whichintroduces the concept of the transformation pipeline as a directedgraph of transformation nodes and messages according to an embodiment ofthe invention.

FIG. 7 is a flow diagram for a transformation pipeline system where theoutput of one data transformation servers as the input of more than onedownstream transformations which introduces the concept of thetransformation pipeline as a directed graph of transformation nodes andmessages according to an embodiment of the invention.

FIG. 8 is a flow diagram for a transformation pipeline system where aset of three data transformations act to form a cyclical pipeline whichalso introduces the concept of the transformation pipeline as a directedgraph of transformation nodes and messages according to an embodiment ofthe invention.

FIG. 9 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

FIG. 10 is a block diagram illustrating an exemplary logicalarchitecture for a client device, according to various embodiments ofthe invention.

FIG. 11 is a block diagram illustrating an exemplary architecturalarrangement of clients, servers, and external services, according tovarious embodiments of the invention.

FIG. 12 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, a highly scalabledistributed connection interface for data capture from multiple networkservice sources.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beunderstood that these are presented for illustrative purposes only. Thedescribed embodiments are not intended to be limiting in any sense. Oneor more of the inventions may be widely applicable to numerousembodiments, as is readily apparent from the disclosure. In general,embodiments are described in sufficient detail to enable those skilledin the art to practice one or more of the inventions, and it is to beunderstood that other embodiments may be utilized and that structural,logical, software, electrical and other changes may be made withoutdeparting from the scope of the particular inventions. Accordingly,those skilled in the art will recognize that one or more of theinventions may be practiced with various modifications and alterations.Particular features of one or more of the inventions may be describedwith reference to one or more particular embodiments or figures thatform a part of the present disclosure, and in which are shown, by way ofillustration, specific embodiments of one or more of the inventions. Itshould be understood, however, that such features are not limited tousage in the one or more particular embodiments or figures withreference to which they are described. The present disclosure is neithera literal description of all embodiments of one or more of theinventions nor a listing of features of one or more of the inventionsthat must be present in all embodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring sequentially (e.g., because one step is described after theother step). Moreover, the illustration of a process by its depiction ina drawing does not imply that the illustrated process is exclusive ofother variations and modifications thereto, does not imply that theillustrated process or any of its steps are necessary to one or more ofthe invention(s), and does not imply that the illustrated process ispreferred. Also, steps are generally described once per embodiment, butthis does not mean they must occur once, or that they may only occuronce each time a process, method, or algorithm is carried out orexecuted. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described, it will be readily apparent that a single deviceor article may be used in place of the more than one device or article.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should be notedthat particular embodiments include multiple iterations of a techniqueor multiple manifestations of a mechanism unless noted otherwise.Process descriptions or blocks in figures should be understood asrepresenting modules, segments, or portions of code which include one ormore executable instructions for implementing specific logical functionsor steps in the process. Alternate implementations are included withinthe scope of embodiments of the present invention in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

Conceptual Architecture

FIG. 1 is a diagram of an exemplary architecture of a distributedoperating system 100 according to an embodiment of the invention. Clientaccess to the system 105 for specific data entry, system control and forinteraction with system output such as automated predictive decisionmaking and planning and alternate pathway simulations, occurs throughthe system's distributed, extensible high bandwidth cloud interface 110which uses a versatile, robust web application driven interface for bothinput and display of client-facing information and a data store 112 suchas, but not limited to MONGODB™, COUCHDB™, CASSANDRA™ or REDIS™depending on the embodiment. Much of the data analyzed by the systemboth from sources within the confines of the client organization, andfrom cloud based sources, also enter the system through the cloudinterface 110, data being passed to the connector module 135 which maypossess the API routines 135 a needed to accept and convert the externaldata and then pass the normalized information to other analysis andtransformation components of the system, the directed computationalgraph module 155, high volume web crawler module 115, multidimensionaltime series database 120 and the graph stack service 145. The directedcomputational graph module 155 retrieves one or more streams of datafrom a plurality of sources, which includes, but is in no way notlimited to, a plurality of physical sensors, network service providers,web based questionnaires and surveys, monitoring of electronicinfrastructure, crowd sourcing campaigns, and human input deviceinformation. Within the directed computational graph module 155, datamay be split into two identical streams in a specialized pre-programmeddata pipeline 155 a, wherein one sub-stream may be sent for batchprocessing and storage while the other sub-stream may be reformatted fortransformation pipeline analysis. The data is then transferred to thegeneral transformer service module 160 for linear data transformation aspart of analysis or the decomposable transformer service module 150 forbranching or iterative transformations that are part of analysis. Thedirected computational graph module 155 represents all data as directedgraphs where the transformations are nodes and the result messagesbetween transformations edges of the graph. The high volume web crawlingmodule 115 uses multiple server hosted preprogrammed web spiders, whichwhile autonomously configured are deployed within a web scrapingframework 115 a of which SCRAPY™ is an example, to identify and retrievedata of interest from web based sources that are not well tagged byconventional web crawling technology. The multiple dimension time seriesdata store module 120 may receive data from a large plurality of sensorsthat may be of several different types. The multiple dimension timeseries data store module may also store any time series data encounteredby the system such as but not limited to component and system logs,performance data, network service information captures such as, but notlimited to news and financial feeds, and sales and service relatedcustomer data. The module is designed to accommodate irregular and highvolume surges by dynamically allotting network bandwidth and serverprocessing channels to process the incoming data. Inclusion ofprogramming wrappers for languages examples of which are, but notlimited to C++, PERL, PYTHON, and ERLANG™ allows sophisticatedprogramming logic to be added to the default function of themultidimensional time series database 120 without intimate knowledge ofthe core programming, greatly extending breadth of function. Dataretrieved by the multidimensional time series database 120 and the highvolume web crawling module 115 may be further analyzed and transformedinto task optimized results by the directed computational graph 155 andassociated general transformer service 150 and decomposable transformerservice 160 modules. Alternately, data from the multidimensional timeseries database and high volume web crawling modules may be sent, oftenwith scripted cuing information determining important vertexes 145 a, tothe graph stack service module 145 which, employing standardizedprotocols for converting streams of information into graphrepresentations of that data, for example, open graph internettechnology although the invention is not reliant on any one standard.Through the steps, the graph stack service module 145 represents data ingraphical form influenced by any pre-determined scripted modifications145 a and stores it in a graph-based data store 145 b such as GIRAPH™ ora key value pair type data store REDIS™, or RIAK™, among others, all ofwhich are suitable for storing graph-based information.

Results of the transformative analysis process may then be combined withfurther client directives, additional rules and practices relevant tothe analysis and situational information external to the alreadyavailable data in the automated planning service module 130 which alsoruns powerful information theory 130 a based predictive statisticsfunctions and machine learning algorithms to allow future trends andoutcomes to be rapidly forecast based upon the current system derivedresults and choosing each a plurality of possible decisions. The usingall available data, the automated planning service module 130 maypropose decisions most likely to result is the most favorable outcomewith a usably high level of certainty. Closely related to the automatedplanning service module in the use of system derived results inconjunction with possible externally supplied additional information inthe assistance of end user decision making, the action outcomesimulation module 125 with its discrete event simulator programmingmodule 125 a coupled with the end user facing observation and stateestimation service 140 which is highly scriptable 140 b as circumstancesrequire and has a game engine 140 a to more realistically stage possibleoutcomes of decisions under consideration, allows decision makers toinvestigate the probable outcomes of choosing one pending course ofaction over another based upon analysis of the current available data.For example, the pipelines operations department has reported a verysmall reduction in crude oil pressure in a section of pipeline in ahighly remote section of territory. Many believe the issue is entirelydue to a fouled, possibly failing flow sensor, others believe that it isa proximal upstream pump that may have foreign material stuck in it.Correction of both of these possibilities is to increase the output ofthe effected pump to hopefully clean out it or the fouled sensor. Afailing sensor will have to be replaced at the next maintenance cycle. Afew, however, feel that the pressure drop is due to a break in thepipeline, probably small at this point, but even so, crude oil isleaking and the remedy for the fouled sensor or pump option could makethe leak much worse and waste much time afterwards. The company doeshave a contractor about 8 hours away, or could rent satellite time tolook but both of those are expensive for a probable sensor issue,significantly less than cleaning up an oil spill though and then withsignificant negative public exposure. These sensor issues have happenedbefore and the distributed operating system 100 has data from them,which no one really studied due to the great volume of columnar figures,so the alternative courses 125, 140 of action are run. The system, basedon all available data, predicts that the fouled sensor or pump isunlikely to be the root cause this time due to other available data, andthe contractor is dispatched. She finds a small breach in the pipeline.There will be a small cleanup and the pipeline needs to be shut down forrepair but multiple tens of millions of dollars have been saved. This isjust one example of a great many of the possible use of the distributedoperating system, those knowledgeable in the art will easily formulatemore.

FIG. 2 is a diagram of an exemplary architecture of a connector moduleand related modules according to an embodiment of the invention 200. Theconnector module 135 may be comprised of a distributed multiserviceconnection module 231 which coordinates connections between thedistributed operating system 100 and external network service sourceswhich may be, for example commercial, cloud based services such as butnot limited to SALESFORCE™, BLOOMBERG™, THOMSON-REUTERS™, TWITTER™,FACEBOOK™, and GOOGLE™, while others may be internal network servicessuch as a wireless network health monitor or applications both internaland external that provide output data required by organizations. Thedistributed multi-service connection module 231 comprises the APIroutines that allow it to retrieve either by passive stream monitoringor time or event driven active retrieval depending on the source and thepre-scripted instructions. API routines, analyst generated scriptsgoverning connector module 135 operation, any needed parameters such assecurity and subscription credentials needed for one or more of thenetwork service, command modifiers, trigger event descriptors, and timeperiod descriptors to list just a few examples may be stored in aparameter data store 233. The inclusion of a robust, expressivescripting language with advanced logic constructs 232 which allows therouting and handling of data within and out of the distributedmulti-service connection module sets this connection interface apartfrom those currently available such as ZAPIER™ and IFTTT™. The abilityto retrieve data on event or time dependent bases raises the connectormodule's 135 abilities above those of SPARKTA™. Of great importance whenusing connection interface similar to that described here, is that evenunexpectedly large influxes of data may be received without loss. Toaccount for these possibilities, the connector module 100 is designedand implemented as a distributed cluster module that is highly andrapidly scalable and the module is self-load balancing capable 234.Information is captured, may have simple transformation done by the APIroutines but may also have more extensive transformation to convert intoforms that are appropriate for the pre-intended use. Much of the dataentering the distributed operating system 100 through the connectormodule 135 may thus be modified by decomposable transformer servicemodule 150, which is accessed through distributed computational graphmodule 155. The decomposable transformer service module 150 may beemployed in these instances because it is able to perform complex seriestransformation pathways which may be simple linear 500, branching 600,two sources into one output 700, and reiterative 800. The nature oftransformations done are completely dependent on the intended downstreamusage of that data with coding for each transformation pre-programmedand pre-selected for those purposes. Data, raw or transformed, mayfollow one of a plurality of output pathways as pre-programmed 232 forthe data source and type. The data may be directly displayed at a clientaccess terminal 210 which may be remote and network connected 220 or maybe directly connected to the system (not shown for simplicity). Timeseries data, including system logs, performance data, and componentlogs, among others may be stored persistently in the multidimensionaltime series data store 120 which is specifically designed and thereforewell suited for such data type. Data, raw or transformed may be storedin another data store 250 within the system as per authorpre-determination or, the data may be sent to other components of thedistributed system 290, 100, for example the automated planning servicemodule 130 for predictive analytics, the action outcome simulationmodule 125 for simulation construction or the observation and stateestimation service 140 for graphical representation.

FIG. 3 is a flow diagram of the operation of an exemplary connectormodule according to an embodiment of the invention 300. Information froma plurality of network or cloud based service source which may includebut are not limited to SALESFORCE™, BLOOMBERG™, THOMSON-REUTERS™,TWITTER™, FACEBOOK™, and GOOGLE™ using a connector module 135specifically designed for the task 302. The connector module may storeand retrieve API routines for the network services from which thedesired information is retrieved as well as other parameters such as anysecurity or subscription credentials, among other task relatedinformation from one or more databases in a data store 301. Retrievalmay occur by passive monitoring of a network service's published datastream as may be the case for sources such as news providers orinvestment market tickers, to name a few such streaming sources known tothose skilled in the art as important to intelligence and operationsthrough the use of predefined filters. Alternatively, retrieval mayoccur from a subset of network service sources on the basis of apre-decided and pre-scripted triggering event of set of triggeringevents or on a timed interval trigger where the source may be polled fornew information either at specific timed intervals or at specific timesof the day. Other triggers for information retrieval may be known tothose skilled in the art and do to robust, expressive python basedscripting language designed into the connector module 135, the inventionmay be configured to employ any such strategy that can be programmedinto a computing device.

Invoking scripts to be employed for specific triggers, time based orevent based is simplified by the use of separate parameter files asample template of which is shown:

01 “triggers”: [ 02 { 03  “uuid”: “abscefg”, 04 “name”: “trigger-name”,05 “type”: “time OR event”, 06  “condition”: [ 07 “name”:“condition-name”, 08 “description”: “condition-description”, 09“pythonToEvaluate”: “function which returns a boolean when specifiedinput is provided” 10 “pythonToFormatOutputForAction”: “function toprocess available data/ state information into action format” 11  ] 12 “action”: “123456”, 13 }

Of special note in the above listing are line 10 where, based upon thetrigger, specific formatting may be performed on the incoming data priorto that data being routed to another module in the system 100 forpossible further processing or display, and line 12 where the nextaction to be performed, most likely by another module of the distributedoperating system such as, but not limited to the digital computationalgraph module 155 and decomposable transformer service module 150 303,the multidimensional time series data store 120, display at a clientaccess terminal 105 or persistent storage in a data store (not shown).Actions brought about by combinations of these and other system modulesas also possible. While other distributed system modules may participatein the processing of information retrieved by the connector module 200,302, Much of the data modification done 303 may require thetransformative capabilities of the decomposable transformer servicemodule 150, which is accessed through distributed computational graphmodule 155, 400. The decomposable transformer service module 150 may beemployed in these instances because it is able to perform complex seriestransformation pathways which may be simple linear 500, branching 600,two sources into one output 700, and reiterative 800. The nature oftransformations done, for example, aggregation or audio to texttranslation are completely dependent on the intended downstream usage ofthat data with coding for each transformation pre-programmed andpre-selected for those purposes. Transformed data may then follow one ofseveral paths to useful disposition which non-exhaustively includespassing the data to other modules of the distributed operating system100, 308, displaying the data in tabular of graphical formats 309, orstoring the data in a data store most suited to the type of datareceived 306, 307. Other activities performed by the connector modulesuch as, but not limited to simple data aggregation and outputformatting and routing are controlled by the same easily generated andmaintained parameter lists and underlying PYTHON™ based scripts aslisted above. It should be noted that, while PYTHON™ is currently usedas the underlying scripting language, the invention is not reliant uponany specific language to fulfill this purpose and any similar scriptinglanguage known to those skilled in the art may be used in its place asutility warrants. Last, each retrieval and processing step, as well assupporting system activities as well as performance data, which may beinvolved in SLA standards compliance may be stored in themultidimensional time series data store 304, 120 either for metric oranalytical monitoring transmission or later inspection duringtroubleshooting or metric review at a later time.

FIG. 4 is a process flow diagram of a method 400 for predictive analysisof very large data sets using the decomposable transformation servicemodule. One or more streams of data from a plurality of sources, whichincludes, but is in no way not limited to, the connector module 135, 200of the distributed operating system 100, a number of physical sensors,web based questionnaires and surveys, monitoring of electronicinfrastructure, crowd sourcing campaigns, and direct human interaction,may be received by system 401. The received stream is filtered 402 toexclude data that has been corrupted, data that is incomplete ormisconfigured and therefore unusable, data that may be intact butnonsensical within the context of the analyses being run, as well as aplurality of predetermined analysis related and unrelated criteria setby the authors. Filtered data may be split into two identical streams atthis point (second stream not depicted for simplicity), wherein onesubstream may be sent for batch processing while another substream maybe formalized 403 for transformation pipeline analysis 404, 500, 600,700, 800. Data formalization for transformation pipeline analysis actsto reformat the stream data for optimal, reliable use during analysis.Reformatting might entail, but is not limited to: setting data fieldorder, standardizing measurement units if choices are given, splittingcomplex information into multiple simpler fields, and stripping unwantedcharacters, again, just to name a few simple examples. The formalizeddata stream may be subjected to one or more transformations. Eachtransformation acts as a function on the data and may or may not changethe data. Within the invention, transformations working on the same datastream where the output of one transformation acts as the input to thenext are represented as transformation pipelines. While the greatmajority of transformations in transformation pipelines receive a singlestream of input, modify the data within the stream in some way and thenpass the modified data as output to the next transformation in thepipeline, the invention does not require these characteristics.According to the embodiment, individual transformations may receiveinput of expected form from more than one source 700 or receive no inputat all as would a transformation acting as a timestamp. According to theembodiment, individual transformations, may not modify the data as wouldbe encountered with a data store acting as a queue for downstreamtransformations described in ¶064 of co-pending application Ser. No.14/925,974. According to the embodiment, individual transformations mayprovide output to more than one downstream transformations 600. Thisability lends itself to simulations where multiple possible choicesmight be made at a single step of a procedure all of which need to beanalyzed. While only a single, simple use case has been offered for eachexample, in each case, that example was chosen for simplicity ofdescription from a plurality of possibilities, the examples given shouldnot be considered to limit the invention to only simplisticapplications. Last, according to the invention, transformations in atransformation pipeline backbone may form a linear, a quasi-lineararrangement or may be cyclical 800, where the output of one of theinternal transformations serves as the input of one of its antecedentsallowing recursive analysis to be run. The result of transformationpipeline analysis may then be modified by results from batch analysis ofthe data stream and output in format predesigned by the authors of theanalysis with could be human readable summary printout, human readableinstruction printout, human-readable raw printout, data store, ormachine encoded information of any format known to the art to be used infurther automated analysis or action schema.

FIG. 5 is a block diagram of a preferred architecture for atransformation pipeline within a system for predictive analysis of verylarge data sets using distributed computational graph 500. According tothe embodiment, streaming input from the data filter software module520, 515 serves as input to the first transformation node 520 of thetransformation pipeline. Transformation node's function is performed oninput data stream and transformed output message 525 is sent totransformation node 2 530. The progression of transformation nodes 520,530, 540, 550, 560 and associated output messages from each node 525,535, 545, 555 is linear in configuration this is the simplestarrangement and, as previously noted, represents the current state ofthe art. While transformation nodes are described according to variousembodiments as uniform shape (referring to FIGS. 5-8), such uniformityis used for presentation simplicity and clarity and does not reflectnecessary operational similarity between transformations within thepipeline. It should be appreciated that one knowledgeable in the fieldwill realize that certain transformations in a pipeline may be entirelyself-contained; certain transformations may involve direct humaninteraction 530, such as selection via dial or dials, positioning ofswitch or switches, or parameters set on control display, all of whichmay change during analysis; other transformations may require externalaggregation or correlation services or may rely on remote procedurecalls to synchronous or asynchronous analysis engines as might occur insimulations among a plurality of other possibilities. Further, accordingto the embodiment, individual transformation nodes in one pipeline mayrepresent function of another transformation pipeline. It should beappreciated that the node length of transformation pipelines depicted inno way confines the transformation pipelines employed by the inventionto an arbitrary maximum length 540, 550, 560 as, being distributed, thenumber of transformations would be limited by the resources madeavailable to each implementation of the invention. It should be furtherappreciated that there need be no limits on transform pipeline length.Output of the last transformation node and by extension, the transformpipeline 560 may be sent back to connector module 200 for predeterminedaction.

FIG. 6 is a block diagram of another preferred architecture for atransformation pipeline within a system for predictive analysis of verylarge data sets using distributed computational graph 600. According tothe embodiment, streaming input from a data filter software module 200,605 serves as input to the first transformation node 610 of thetransformation pipeline. Transformation node's function is performed oninput data stream and transformed output message 615 is sent totransformation node 2 620. In this embodiment, transformation node 2 620has a second input stream 660. The specific source of this input isinconsequential to the operation of the invention and could be anothertransformation pipeline software module, a data store, humaninteraction, physical sensors, monitoring equipment for other electronicsystems or a stream from the internet as from a crowdsourcing campaign,just to name a few possibilities 660. Functional integration of a secondinput stream into one transformation node requires the two input streamevents be serialized. The invention performs this serialization using adecomposable transformation software module. While transformation nodesare described according to various embodiments as uniform shape(referring to FIGS. 5-8), such uniformity is used for presentationsimplicity and clarity and does not reflect necessary operationalsimilarity between transformations within the pipeline. It should beappreciated that one knowledgeable in the field will realize thatcertain transformations in a pipeline may be entirely self-contained;certain transformations may involve direct human interaction 630, suchas selection via dial or dials, positioning of switch or switches, orparameters set on control display, all of which may change duringanalysis; other transformations may require external aggregation orcorrelation services or may rely on remote procedure calls tosynchronous or asynchronous analysis engines as might occur insimulations among a plurality of other possibilities. Further accordingto the embodiment, individual transformation nodes in one pipeline mayrepresent function of another transformation pipeline. It should beappreciated that the node length of transformation pipelines depicted inno way confines the transformation pipelines employed by the inventionto an arbitrary maximum length 610, 620, 630, 640, 650, as, beingdistributed, the number of transformations would be limited by theresources made available to each implementation of the invention. Itshould be further appreciated that there need be no limits on transformpipeline length. Output of the last transformation node and byextension, the transform pipeline, 650 may be sent back to connectormodule 200 for pre-decided action.

FIG. 7 is a block diagram of another preferred architecture for atransformation pipeline within a system for predictive analysis of verylarge data sets using distributed computational graph 700. According tothe embodiment, streaming input from a data filter software module 300,705 serves as input to the first transformation node 710 of thetransformation pipeline. Transformation node's function is performed oninput data stream and transformed output message 715 is sent totransformation node 2 720. In this embodiment, transformation node 2 720sends its output stream to two transformation pipelines 730, 740, 750,765, 775. This allows the same data stream to undergo two disparate,possibly completely unrelated, analyses without having to duplicate theinfrastructure of the initial transform manipulations, greatlyincreasing the expressivity of the invention over current transformpipelines. Functional integration of a second output stream from onetransformation node 720 requires that the two output stream events beserialized. The invention performs this serialization using adecomposable transformation software module 150. While transformationnodes are described according to various embodiments as uniform shape(referring to FIGS. 5-8), such uniformity is used for presentationsimplicity and clarity and does not reflect necessary operationalsimilarity between transformations within the pipeline. It should beappreciated that one knowledgeable in the field will realize thatcertain transformations in pipelines, which may be entirelyself-contained; certain transformations may involve direct humaninteraction, such as selection via dial or dials, positioning of switchor switches, or parameters set on control display, all of which maychange during analysis; other transformations may require externalaggregation or correlation services or may rely on remote procedurecalls to synchronous or asynchronous analysis engines as might occur insimulations, among a plurality of other possibilities. Further accordingto the embodiment, individual transformation nodes in one pipeline mayrepresent function of another transformation pipeline. It should beappreciated that the node number of transformation pipelines depicted inno way confines the transformation pipelines employed by the inventionto an arbitrary maximum length 710, 720, 730, 740, 750; 765, 775 as,being distributed, the number of transformations would be limited by theresources made available to each implementation of the invention.Further according to the embodiment, there need be no limits ontransform pipeline length. Output of the last transformation node and byextension, the transform pipeline 750 may be sent back to connectormodule 135 for programmatically enabled action.

FIG. 8 is a block diagram of another preferred architecture for atransformation pipeline within a system for predictive analysis of verylarge data sets using distributed computational graph 700. According tothe embodiment, streaming input from a data filter software module 520,805 serves as input to the first transformation node 810 of thetransformation pipeline. Transformation node's function may be performedon an input data stream and transformed output message 815 may then besent to transformation node 2 820. Likewise, once the data stream isacted upon by transformation node 2 820, its output is sent totransformation node 3 830 using its output message 825 In thisembodiment, transformation node 3 830 sends its output stream back totransform node 1 810 forming a cyclical relationship betweentransformation nodes 1 810, transformation node 2 820 and transformationnode 3 830. Upon the achievement of some gateway result, the output ofcyclical pipeline activity may be sent to downstream transformationnodes within the pipeline 840, 845. The presence of a generalizedcyclical pathway construct allows the invention to be used to solvecomplex iterative problems with large data sets involved, expandingability to rapidly retrieve conclusions for complicated issues.Functional creation of a cyclical transformation pipeline requires thateach cycle be serialized. The invention performs this serializationusing a decomposable transformation software module, the function ofwhich is fully described in ¶065 and ¶066 of co-pending application Ser.No. 14/925,974. While transformation nodes are described according tovarious embodiments as uniform shape (referring to FIGS. 5-8), suchuniformity is used for presentation simplicity and clarity and does notreflect necessary operational similarity between transformations withinthe pipeline. It should be appreciated that one knowledgeable in thefield will appreciate that certain transformations in pipelines, may beentirely self-contained; certain transformations may involve directhuman interaction 530, such as selection via dial or dials, positioningof switch or switches, or parameters set on control display, all ofwhich may change during analysis; still other transformations mayrequire external aggregation or correlation services or may rely onremote procedure calls to synchronous or asynchronous analysis enginesas might occur in simulations, among a plurality of other possibilities.Further according to the embodiment, individual transformation nodes inone pipeline may represent the cumulative function of anothertransformation pipeline. It should be appreciated that the node numberof transformation pipelines depicted in no way confines thetransformation pipelines employed by the invention to an arbitrarymaximum length 810, 820, 830, 840, 850; 865, 875 as, being distributed,the number of transformations would be limited by the resources madeavailable to each implementation of the invention. It should be furtherappreciated that there need be no limits on transform pipeline length.Output of the last transformation node and by extension, the transformpipeline 855 may be sent back to connector module 200 forprogrammatically enabled action.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be describedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, or other appropriatecomputing device), a consumer electronic device, a music player, or anyother suitable electronic device, router, switch, or other suitabledevice, or any combination thereof. In at least some embodiments, atleast some of the features or functionalities of the various embodimentsdisclosed herein may be implemented in one or more virtualized computingenvironments (e.g., network computing clouds, virtual machines hosted onone or more physical computing machines, or other appropriate virtualenvironments).

Referring now to FIG. 9, there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one embodiment, a computing device 10 may beconfigured or designed to function as a server system utilizing CPU 12,local memory 11 and/or remote memory 16, and interface(s) 15. In atleast one embodiment, CPU 12 may be caused to perform one or more of thedifferent types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 13 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a specific embodiment,a local memory 11 (such as non-volatile random access memory (RAM)and/or read-only memory (ROM), including for example one or more levelsof cached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QualcommSNAPDRAGON™ or Samsung EXYNOS™ CPU as are becoming increasingly commonin the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 15 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity A/V hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown and described above illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 13 handles communicationsas well as routing computations, while in other embodiments a separatededicated communications processor may be provided. In variousembodiments, different types of features or functionalities may beimplemented in a system according to the invention that includes aclient device (such as a tablet device or smartphone running clientsoftware) and server systems (such as a server system described in moredetail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 16 and local memory 11) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 16 or memories 11,16 may also be configured to store data structures, configuration data,encryption data, historical system operations information, or any otherspecific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 10,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 20 includes processors 21that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 24. Processors 21 may carry out computinginstructions under control of an operating system 22 such as, forexample, a version of Microsoft's WINDOWS™ operating system, Apple's MacOS/X or iOS operating systems, some variety of the Linux operatingsystem, Google's ANDROID™ operating system, or the like. In many cases,one or more shared services 23 may be operable in system 20, and may beuseful for providing common services to client applications 24. Services23 may for example be WINDOWS™ services, user-space common services in aLinux environment, or any other type of common service architecture usedwith operating system 21. Input devices 28 may be of any type suitablefor receiving user input, including for example a keyboard, touchscreen,microphone (for example, for voice input), mouse, touchpad, trackball,or any combination thereof. Output devices 27 may be of any typesuitable for providing output to one or more users, whether remote orlocal to system 20, and may include for example one or more screens forvisual output, speakers, printers, or any combination thereof. Memory 25may be random-access memory having any structure and architecture knownin the art, for use by processors 21, for example to run software.Storage devices 26 may be any magnetic, optical, mechanical, memristor,or electrical storage device for storage of data in digital form (suchas those described above). Examples of storage devices 26 include flashmemory, magnetic hard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 11, there is shown a blockdiagram depicting an exemplary architecture 30 for implementing at leasta portion of a system according to an embodiment of the invention on adistributed computing network. According to the embodiment, any numberof clients 33 may be provided. Each client 33 may run software forimplementing client-side portions of the present invention; clients maycomprise a system 20 such as that illustrated above. In addition, anynumber of servers 32 may be provided for handling requests received fromone or more clients 33. Clients 33 and servers 32 may communicate withone another via one or more electronic networks 31, which may be invarious embodiments any of the Internet, a wide area network, a mobiletelephony network (such as CDMA or GSM cellular networks), a wirelessnetwork (such as WiFi, Wimax, LTE, and so forth), or a local areanetwork (or indeed any network topology known in the art; the inventiondoes not prefer any one network topology over any other). Networks 31may be implemented using any known network protocols, including forexample wired and/or wireless protocols.

In addition, in some embodiments, servers 32 may call external services37 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 37 may take place, for example, via one or morenetworks 31. In various embodiments, external services 37 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 24 are implemented on a smartphone or other electronicdevice, client applications 24 may obtain information stored in a serversystem 32 in the cloud or on an external service 37 deployed on one ormore of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 33 or servers 32 (or both)may make use of one or more specialized services or appliances that maybe deployed locally or remotely across one or more networks 31. Forexample, one or more databases 34 may be used or referred to by one ormore embodiments of the invention. It should be understood by one havingordinary skill in the art that databases 34 may be arranged in a widevariety of architectures and using a wide variety of data access andmanipulation means. For example, in various embodiments one or moredatabases 34 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, Hadoop Cassandra, Google BigTable, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 36 and configuration systems 35. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 36 orconfiguration system 35 or approach is specifically required by thedescription of any specific embodiment.

FIG. 12 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to keyboard 49, pointing device 50,hard disk 52, and real-time clock 51. NIC 53 connects to network 54,which may be the Internet or a local network, which local network may ormay not have connections to the Internet. Also shown as part of system40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for highly-scalable distributedconnection interface for data capture from multiple network servicesources comprising: a connector module comprising connections to aplurality of network data sources and a plurality of programminginstructions configured to operate on a processor of a first computingdevice causing the first computing device to: retrieve a plurality ofdata from each of the plurality of network data sources; accept aplurality of analysis parameters and control commands from a user of thesystem, the plurality of analysis parameters and control commandscomprising: instructions to either passively monitor data or activelyretrieve the plurality of data from each of the plurality of networkdata sources, depending on the each of the plurality network datasources; and send the plurality of data and the plurality of analysisparameters and control commands to a directed computational graphmodule; the directed computational graph module comprising a directedcomputational graph comprising nodes representing data transformationsand edges representing transformed message outputs, and a plurality ofprogramming instructions configured to operate on a processor of asecond computing device causing the second computing device to: receivethe plurality of data and the plurality of analysis parameters andcontrol commands from the connector module; direct a decomposabletransformer service module to convert the plurality of data into aformat; receive data transformations and transformed message outputsfrom the decomposable transformer service module; and update the nodesof the directed computational graph to represent the received datatransformations, and update the edges of the directed computationalgraph to represent the received transformed message outputs; and thedecomposable transformer service module comprising a plurality ofprogramming instructions configured to operate on a processor of a thirdcomputing device causing the third computing device to: receive theplurality of data and direction from the directed computational graphmodule; perform a plurality of transformations on the data, wherein eachof the plurality of transformations receives an input data stream andproduces the transformed message output; and send each datatransformation and transformed message output to the directedcomputational graph module.
 2. The system of claim 1, wherein theplurality of data is retrieved by continuous monitoring by applicationprogramming interface routines of information streams released by theplurality of network data sources.
 3. The system of claim 2, wherein atleast a portion of the plurality of data may be isolated based upon useof filters.
 4. The system of claim 1, wherein at least a portion of theplurality of data is retrieved from the plurality of network datasources based upon an event trigger.
 5. The system of claim 1, whereinat least a portion of the plurality of data is retrieved from theplurality of network data sources based upon a time-dependent trigger.6. The system of claim 1, wherein at least a portion of the data istransformed by the connector module into a format useful for apredetermined purpose.
 7. The system of claim 1, wherein at least aportion of the plurality of data is displayed and discarded.
 8. Thesystem of claim 1, wherein at least a portion of the plurality of datais persistently stored.
 9. A method for highly scalable distributedconnection interface for data capture from multiple network servicesources comprising the steps of: retrieving a plurality of data from aplurality of network data sources; accepting a plurality of analysisparameters and control commands from a user, the plurality of analysisparameters and control commands comprising instructions to eitherpassively monitor data or actively retrieve the plurality of data fromeach of the plurality of network data sources, depending on the each ofthe plurality of network data sources; directing a decomposabletransformer service module of a computing device to convert theplurality of data into a format; converting the plurality of data intothe format by performing a plurality of transformations on the pluralityof data, wherein each of the plurality of transformations receives aninput data stream and produces a transformed message output; andupdating a directed computational graph with the converted data, wherethe directed computational graph comprises nodes which represent theplurality of transformations and edges which represent the transformedmessage outputs.
 10. The method of claim 9, wherein the plurality ofdata is retrieved by continuous monitoring by application programminginterface routines of information streams released by the plurality ofnetwork data sources.
 11. The method of claim 9, wherein at least aportion of the plurality of data may be isolated based upon use offilters.
 12. The method of claim 9, wherein at least a portion of theplurality of data is retrieved from the plurality of network datasources based upon an event trigger.
 13. The method of claim 9, whereinat least a portion of the plurality of data is retrieved from theplurality of network data sources based upon a time dependent trigger.14. The method of claim 9, wherein at least a portion of the pluralityof data is displayed and discarded.
 15. The method of claim 9, whereinat least a portion of the plurality of data is persistently stored.